Evaluation of bias and gender/racial concordance based on sentiment analysis of narrative evaluations of clinical clerkships using natural language processing.

There is increasing interest in understanding potential bias in medical education. We used natural language processing (NLP) to evaluate potential bias in clinical clerkship evaluations. Data from medical evaluations and administrative databases for medical students enrolled in third-year clinical clerkship rotations across two academic years. We collected demographic information of students and faculty evaluators to determine gender/racial concordance (i.e., whether the student and faculty identified with the same demographic). We used a multinomial log-linear model for final clerkship grades, using predictors such as numerical evaluation scores, gender/racial concordance, and sentiment scores of narrative evaluations using the SentimentIntensityAnalyzer tool in Python. 2037 evaluations from 198 students were analyzed. Statistical significance was defined as P < 0.05. Sentiment scores for evaluations did not vary significantly by student gender, race, or ethnicity (P = 0.88, 0.64, and 0.06, respectively). Word choices were similar across faculty and student demographic groups. Modeling showed narrative evaluation sentiment scores were not predictive of an honors grade (odds ratio [OR] 1.23, P = 0.58). Numerical evaluation average (OR 1.45, P < 0.001) and gender concordance between faculty and student (OR 1.32, P = 0.049) were significant predictors of receiving honors. The lack of disparities in narrative text in our study contrasts with prior findings from other institutions. Ongoing efforts include comparative analyses with other institutions to understand what institutional factors may contribute to bias. NLP enables a systematic approach for investigating bias. The insights gained from the lack of association between word choices, sentiment scores, and final grades show potential opportunities to improve feedback processes for students.

'You don't see what I see': Co-designing simulation to uncover and address cognitive bias in healthcare.

EDUCATIONAL CHALLENGE: Each year, adverse events are reported in healthcare, of which many relate to healthcare workforce cognitive bias. The active involvement of workforce and consumers in the review and co-design of effective training for the healthcare workforce to recognise, monitor, and manage unconscious bias is required. PROPOSED SOLUTION: We used participatory action research to co-design an innovative, interprofessional simulation based on 'real world' clinical incidents and lived experiences to improve the delivery of safe, high quality, consumer-focused healthcare. Following ethics approval, content analysis of serious adverse patient safety events involving cognitive bias was conducted. These data informed audio-recorded interviews with the healthcare workforce and consumers to explore their experiences of cognitive bias. Following thematic analysis, key themes of communication, stigma, diagnostic overshadowing, and fragmented systems were uncovered. Guided by consumers, these themes were interwoven into a simulation scenario that included real places, stories, and verbatim quotes delivered through mixed media artefacts. This heightened the immersive and experiential learning that aimed to uncover unconscious bias and help learners recognise its impact on clinical decisions and practice. POTENTIAL BENEFITS AND NEXT STEPS: To our knowledge, this is the first interprofessional, co-designed simulation to specifically address cognitive bias in current and future healthcare workforce. Plans to translate this research into a practical framework on how to work with key stakeholders (including consumers) to identify 'real-world' health service risks and co-design targeted simulations to address these gaps are described, including lessons learned.

A scoping review of artificial intelligence in medical education: BEME Guide No. 84.

BACKGROUND: Artificial Intelligence (AI) is rapidly transforming healthcare, and there is a critical need for a nuanced understanding of how AI is reshaping teaching, learning, and educational practice in medical education. This review aimed to map the literature regarding AI applications in medical education, core areas of findings, potential candidates for formal systematic review and gaps for future research. METHODS: This rapid scoping review, conducted over 16 weeks, employed Arksey and O'Malley's framework and adhered to STORIES and BEME guidelines. A systematic and comprehensive search across PubMed/MEDLINE, EMBASE, and MedEdPublish was conducted without date or language restrictions. Publications included in the review spanned undergraduate, graduate, and continuing medical education, encompassing both original studies and perspective pieces. Data were charted by multiple author pairs and synthesized into various thematic maps and charts, ensuring a broad and detailed representation of the current landscape. RESULTS: The review synthesized 278 publications, with a majority (68%) from North American and European regions. The studies covered diverse AI applications in medical education, such as AI for admissions, teaching, assessment, and clinical reasoning. The review highlighted AI's varied roles, from augmenting traditional educational methods to introducing innovative practices, and underscores the urgent need for ethical guidelines in AI's application in medical education. CONCLUSION: The current literature has been charted. The findings underscore the need for ongoing research to explore uncharted areas and address potential risks associated with AI use in medical education. This work serves as a foundational resource for educators, policymakers, and researchers in navigating AI's evolving role in medical education. A framework to support future high utility reporting is proposed, the FACETS framework.

An Automated System for Physician Trainee Procedure Logging via Electronic Health Records.

Importance: Procedural proficiency is a core competency for graduate medical education; however, procedural reporting often relies on manual workflows that are duplicative and generate data whose validity and accuracy are difficult to assess. Failure to accurately gather these data can impede learner progression, delay procedures, and negatively impact patient safety. Objective: To examine accuracy and procedure logging completeness of a system that extracts procedural data from an electronic health record system and uploads these data securely to an application used by many residency programs for accreditation. Design, Setting, and Participants: This quality improvement study of all emergency medicine resident physicians at University of California, San Diego Health was performed from May 23, 2023, to June 25, 2023. Exposures: Automated system for procedure data extraction and upload to a residency management software application. Main Outcomes and Measures: The number of procedures captured by the automated system when running silently compared with manually logged procedures in the same timeframe, as well as accuracy of the data upload. Results: Forty-seven residents participated in the initial silent assessment of the extraction component of the system. During a 1-year period (May 23, 2022, to May 7, 2023), 4291 procedures were manually logged by residents, compared with 7617 procedures captured by the automated system during the same period, representing a 78% increase. During assessment of the upload component of the system (May 8, 2023, to June 25, 2023), a total of 1353 procedures and patient encounters were evaluated, with the system operating with a sensitivity of 97.4%, specificity of 100%, and overall accuracy of 99.5%. Conclusions and Relevance: In this quality improvement study of emergency medicine resident physicians, an automated system demonstrated that reliance on self-reported procedure logging resulted in significant procedural underreporting compared with the use of data obtained at the point of performance. Additionally, this system afforded a degree of reliability and validity heretofore absent from the usual after-the-fact procedure logging workflows while using a novel application programming interface-based approach. To our knowledge, this system constitutes the first generalizable implementation of an automated solution to a problem that has existed in graduate medical education for decades.

An end-to-end workflow for multiplexed image processing and analysis.

Multiplexed imaging enables the simultaneous spatial profiling of dozens of biological molecules in tissues at single-cell resolution. Extracting biologically relevant information, such as the spatial distribution of cell phenotypes from multiplexed tissue imaging data, involves a number of computational tasks, including image segmentation, feature extraction and spatially resolved single-cell analysis. Here, we present an end-to-end workflow for multiplexed tissue image processing and analysis that integrates previously developed computational tools to enable these tasks in a user-friendly and customizable fashion. For data quality assessment, we highlight the utility of napari-imc for interactively inspecting raw imaging data and the cytomapper R/Bioconductor package for image visualization in R. Raw data preprocessing, image segmentation and feature extraction are performed using the steinbock toolkit. We showcase two alternative approaches for segmenting cells on the basis of supervised pixel classification and pretrained deep learning models. The extracted single-cell data are then read, processed and analyzed in R. The protocol describes the use of community-established data containers, facilitating the application of R/Bioconductor packages for dimensionality reduction, single-cell visualization and phenotyping. We provide instructions for performing spatially resolved single-cell analysis, including community analysis, cellular neighborhood detection and cell-cell interaction testing using the imcRtools R/Bioconductor package. The workflow has been previously applied to imaging mass cytometry data, but can be easily adapted to other highly multiplexed imaging technologies. This protocol can be implemented by researchers with basic bioinformatics training, and the analysis of the provided dataset can be completed within 5-6 h. An extended version is available at https://bodenmillergroup.github.io/IMCDataAnalysis/ .

Students' Perspectives on Basic and Clinical Science Integration When Step 1 is Administered After the Core Clerkships.

Phenomenon: According to adult learning theories, effective cognitive integration of basic and clinical science may promote the transfer of knowledge to patient care. The placement of the U.S. Medical Licensing Examination (USMLE) Step 1 after the core clerkships is one strategy intended to facilitate cognitive integration, though learner experiences with this model are unexplored. The purpose of this study is to understand students' perspectives on basic and clinical science integration in a post-clerkship Step 1 curriculum. Approach: Focus groups were conducted between August and September 2020 with senior medical students from the University of California, San Francisco School of Medicine and University of Michigan Medical School. Data were analyzed using a constructivist approach to thematic analysis. Findings: Thirty-three students participated in six focus groups. Participants described multiple barriers to cognitive integration in the clerkship learning environment, though they also identified examples of teaching and learning that facilitated integration. Early in their clerkships, students struggled to integrate because of their tenuous basic science foundation, cognitive overload, and difficulty perceiving the relevance of basic science to patient care. They felt that educators primarily focused on clinical science, and many basic science teaching sessions during clerkships felt irrelevant to patient care. However, students also described experiences that made the connection between basic and clinical science more explicit, including modeling by educators and clerkship learning activities that more overtly encouraged the application of basic science to clinical care. In addition, the return to basic science studying during the post-clerkship dedicated Step 1 study period offered powerful integration opportunities. These facilitators of cognitive integration helped students recognize the value of integration for enduring learning. Insights: There are myriad barriers to cognitive integration of basic and clinical science during clerkships in a post-clerkship Step 1 curriculum. The relevance of basic science to patient care needs to be made more explicit to students through modeling by clinician educators to augment the potential benefits of curricular change. The post-clerkship Step 1 study period appears to offer a unique opportunity for cognitive integration later in a learner's trajectory that may be related to curricular design. When learners recognize the applicability of basic science to patient care, they may more intentionally transfer basic science knowledge to clinical practice.

A scoping review of distributed cognition in acute care clinical decision-making.

OBJECTIVES: In acute care settings, interactions between providers and tools drive clinical decision-making. Most studies of decision-making focus on individual cognition and fail to capture critical collaborations. Distributed Cognition (DCog) theory provides a framework for examining the dispersal of tasks among agents and artifacts, enhancing the investigation of decision-making and error. CONTENT: This scoping review maps the evidence collected in empiric studies applying DCog to clinical decision-making in acute care settings and identifies gaps in the existing literature. SUMMARY AND OUTLOOK: Thirty-seven articles were included. The majority (n=30) used qualitative methods (observations, interviews, artifact analysis) to examine the work of physicians (n=28), nurses (n=27), residents (n=16), and advanced practice providers (n=12) in intensive care units (n=18), operating rooms (n=7), inpatient units (n=7) and emergency departments (n=5). Information flow (n=30) and task coordination (n=30) were the most frequently investigated elements of DCog. Provider-artifact (n=35) and provider-provider (n=30) interactions were most explored. Electronic (n=18) and paper (n=15) medical records were frequently described artifacts. Seven prominent themes were identified. DCog is an underutilized framework for examining how information is obtained, represented, and transmitted through complex clinical systems. DCog offers mechanisms for exploring how technologies, like EMRs, and workspaces can help or hinder clinical decision-making.

Virtual interviewing for graduate medical education recruitment and selection: A BEME systematic review: BEME Guide No. 80.

BACKGROUND: The COVID-19 pandemic caused graduate medical education (GME) programs to pivot to virtual interviews (VIs) for recruitment and selection. This systematic review synthesizes the rapidly expanding evidence base on VIs, providing insights into preferred formats, strengths, and weaknesses. METHODS: PubMed/MEDLINE, Scopus, ERIC, PsycINFO, MedEdPublish, and Google Scholar were searched from 1 January 2012 to 21 February 2022. Two authors independently screened titles, abstracts, full texts, performed data extraction, and assessed risk of bias using the Medical Education Research Quality Instrument. Findings were reported according to Best Evidence in Medical Education guidance. RESULTS: One hundred ten studies were included. The majority (97%) were from North America. Fourteen were conducted before COVID-19 and 96 during the pandemic. Studies involved both medical students applying to residencies (61%) and residents applying to fellowships (39%). Surgical specialties were more represented than other specialties. Applicants preferred VI days that lasted 4-6 h, with three to five individual interviews (15-20 min each), with virtual tours and opportunities to connect with current faculty and trainees. Satisfaction with VIs was high, though both applicants and programs found VIs inferior to in-person interviews for assessing 'fit.' Confidence in ranking applicants and programs was decreased. Stakeholders universally noted significant cost and time savings with VIs, as well as equity gains and reduced carbon footprint due to eliminating travel. CONCLUSIONS: The use of VIs for GME recruitment and selection has accelerated rapidly. The findings of this review offer early insights that can guide future practice, policy, and research.

Remote learning developments in postgraduate medical education in response to the COVID-19 pandemic - A BEME systematic review: BEME Guide No. 71.

BACKGROUND: Prior reviews investigated medical education developments in response to COVID-19, identifying the pivot to remote learning as a key area for future investigation. This review synthesized online learning developments aimed at replacing previously face-to-face 'classroom' activities for postgraduate learners. METHODS: Four online databases (CINAHL, Embase, PsychINFO, and PubMed) and MedEdPublish were searched through 21 December 2020. Two authors independently screened titles, abstracts and full texts, performed data extraction, and assessed risk of bias. The PICRAT technology integration framework was applied to examine how teachers integrated and learners engaged with technology. A descriptive synthesis and outcomes were reported. A thematic analysis explored limitations and lessons learned. RESULTS: Fifty-one publications were included. Fifteen collaborations were featured, including international partnerships and national networks of program directors. Thirty-nine developments described pivots of existing educational offerings online and twelve described new developments. Most interventions included synchronous activities (n Fif5). Virtual engagement was promoted through chat, virtual whiteboards, polling, and breakouts. Teacher's use of technology largely replaced traditional practice. Student engagement was largely interactive. Underpinning theories were uncommon. Quality assessments revealed moderate to high risk of bias in study reporting and methodology. Forty-five developments assessed reaction; twenty-five attitudes, knowledge or skills; and two behavior. Outcomes were markedly positive. Eighteen publications reported social media or other outcomes, including reach, engagement, and participation. Limitations included loss of social interactions, lack of hands-on experiences, challenges with technology and issues with study design. Lessons learned highlighted the flexibility of online learning, as well as practical advice to optimize the online environment. CONCLUSIONS: This review offers guidance to educators attempting to optimize learning in a post-pandemic world. Future developments would benefit from leveraging collaborations, considering technology integration frameworks, underpinning developments with theory, exploring additional outcomes, and designing and reporting developments in a manner that supports replication.

Online learning developments in undergraduate medical education in response to the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 69.

BACKGROUND: The COVID-19 pandemic spurred an abrupt transition away from in-person educational activities. This systematic review investigated the pivot to online learning for nonclinical undergraduate medical education (UGME) activities and explored descriptions of educational offerings deployed, their impact, and lessons learned. METHODS: The authors systematically searched four online databases and conducted a manual electronic search of MedEdPublish up to December 21, 2020. Two authors independently screened titles, abstracts and full texts, performed data extraction and assessed risk of bias. A third author resolved discrepancies. Findings were reported in accordance with the STORIES (STructured apprOach to the Reporting in healthcare education of Evidence Synthesis) statement and BEME guidance. RESULTS: Fifty-six articles were included. The majority (n = 41) described the rapid transition of existing offerings to online formats, whereas fewer (n = 15) described novel activities. The majority (n = 27) included a combination of synchronous and asynchronous components. Didactics (n = 40) and small groups (n = 26) were the most common instructional methods. Teachers largely integrated technology to replace and amplify rather than transform learning, though learner engagement was often interactive. Thematic analysis revealed unique challenges of online learning, as well as exemplary practices. The quality of study designs and reporting was modest, with underpinning theory at highest risk of bias. Virtually all studies (n = 54) assessed reaction/satisfaction, fewer than half (n = 23) assessed changes in attitudes, knowledge or skills, and none assessed behavioral, organizational or patient outcomes. CONCLUSIONS: UGME educators successfully transitioned face-to-face instructional methods online and implemented novel solutions during the COVID-19 pandemic. Although technology's potential to transform teaching is not yet fully realized, the use of synchronous and asynchronous formats encouraged virtual engagement, while offering flexible, self-directed learning. As we transition from emergency remote learning to a post-pandemic world, educators must underpin new developments with theory, report additional outcomes and provide details that support replication.

Pivot to online learning for adapting or continuing workplace-based clinical learning in medical education following the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 70.

BACKGROUND: The novel coronavirus disease was declared a pandemic in March 2020, which necessitated adaptations to medical education. This systematic review synthesises published reports of medical educational developments and innovations that pivot to online learning from workplace-based clinical learning in response to the pandemic. The objectives were to synthesise what adaptations/innovation were implemented (description), their impact (justification), and 'how' and 'why' these were selected (explanation and rationale). METHODS: The authors systematically searched four online databases up to December 21, 2020. Two authors independently screened titles, abstracts and full-texts, performed data extraction, and assessed the risk of bias. Our findings are reported in alignment with the STORIES (STructured apprOach to the Reporting in healthcare education of Evidence Synthesis) statement and BEME guidance. RESULTS: Fifty-five articles were included. Most were from North America (n = 40), and nearly 70% focused on undergraduate medical education (UGME). Key developments were rapid shifts from workplace-based learning to virtual spaces, including online electives, telesimulation, telehealth, radiology, and pathology image repositories, live-streaming or pre-recorded videos of surgical procedures, stepping up of medical students to support clinical services, remote adaptations for clinical visits, multidisciplinary team meetings and ward rounds. Challenges included lack of personal interactions, lack of standardised telemedicine curricula and need for faculty time, technical resources, and devices. Assessment of risk of bias revealed poor reporting of underpinning theory, resources, setting, educational methods, and content. CONCLUSIONS: This review highlights the response of medical educators in deploying adaptations and innovations. Whilst few are new, the complexity, concomitant use of multiple methods and the specific pragmatic choices of educators offers useful insight to clinical teachers who wish to deploy such methods within their own practice. Future works that offer more specific details to allow replication and understanding of conceptual underpinnings are likely to justify an update to this review.

Dealing with uncertainty in clinical reasoning: A threshold model and the roles of experience and task framing.

INTRODUCTION: Uncertainty is integral to clinical practice and clinical reasoning but has proven difficult to study and model. Little is known about how clinicians manage uncertainty. According to evidence-based medicine theory, clinicians should utilise new information to reduce uncertainty until reaching action thresholds for further information gathering or treatment. We examined the impact of experience and task framing on uncertainty thresholds and the extent to which these thresholds guided clinical decisions. Finally, we sought to determine the impact of framing by having participants provide threshold responses as a range or as specific numbers. METHODS: One hundred sixty-eight fourth-year medical students, 93 residents and 72 faculty were presented a case of viral pneumonia with a suspected superimposed bacterial infection. Participants identified their testing and treatment thresholds with either a specific number or an inter-threshold range of probabilities that would compel them to test further. Afterwards, they were told the patient had a 20% pre-test probability of a superimposed infection and asked whether they would treat the patient with antibiotics, order additional testing or neither. Responses were compared with their previously stated threshold values to assess decision-making consistency. RESULTS: Testing thresholds were 15.8%, 20.6% and 25.8%, treatment thresholds were 78.5%, 71.6% and 73.4% and threshold spans (difference between testing and treatment thresholds) were 62.7, 51 and 47.6 for students, residents and faculty, respectively. Sixty-four percent of respondents made judgements consistent with their thresholds, 28% escalated their decision (doing more than their thresholds predicted) and 7.6% de-escalated their decision (doing less than their thresholds predicted). Framing had an impact on both faculty and resident decisions and a larger impact on students. DISCUSSION: These findings help us understand how clinical reasoning and threshold determinations vary with clinical experience. As uncertainty can lead to unnecessary testing and cognitive discomfort, examining decision thresholds helps us ascertain how diagnostic and treatment decisions are made.

Understanding diagnostic processes in emergency departments: a mixed methods case study protocol.

INTRODUCTION: Diagnostic processes in the emergency department (ED) involve multiple interactions among individuals who interface with information systems to access and record information. A better understanding of diagnostic processes is needed to mitigate errors. This paper describes a study protocol to map diagnostic processes in the ED as a foundation for developing future error mitigation strategies. METHODS AND ANALYSIS: This study of an adult and a paediatric academic ED uses a prospective mixed methods case study design informed by an ED-specific diagnostic decision-making model (the modified ED-National Academies of Sciences, Engineering and Medicine (NASEM) model) and two cognitive theories (dual process theory and distributed cognition). Data sources include audio recordings of patient and care team interactions, electronic health record data, observer field notes and stakeholder interviews. Multiple qualitative analysis methods will be used to explore diagnostic processes in situ, including systems information flow, human-human and human-system interactions and contextual factors influencing cognition. The study has three parts. Part 1 involves prospective field observations of patients with undifferentiated symptoms at high risk for diagnostic error, where each patient is followed throughout the entire care delivery process. Part 2 involves observing individual care team providers over a 4-hour window to capture their diagnostic workflow, team coordination and communication across multiple patients. Part 3 uses interviews with key stakeholders to understand different perspectives on the diagnostic process, as well as perceived strengths and vulnerabilities, in order to enrich the ED-NASEM diagnostic model. ETHICS AND DISSEMINATION: The University of Michigan Institutional Review Board approved this study, HUM00156261. This foundational work will help identify strengths and vulnerabilities in diagnostic processes. Further, it will inform the future development and testing of patient, provider and systems-level interventions for mitigating error and improving patient safety in these and other EDs. The work will be disseminated through journal publications and presentations at national and international meetings.

The Optimal Timing of Step 1 in Medical Education Following the Transition to Pass/Fail: A Unique Perspective from Post-clerkship Step 1 Schools.

The National Board of Medical Examiners' decision to change Step 1 of the United States Medical Licensing Examination (USMLE) from a three-digit score to Pass/Fail (P/F) represents a disruptive change for students, faculty, and leaders in the academic community. In the context of this change, some schools may re-consider the optimal timing of Step 1 as they strive to align their assessment practices with sound educational principles. Currently, over 20 schools administer USMLE Step 1 after the core clerkships. In this commentary, we review the educational rationale for a post-clerkship Step 1, highlighting how adult learning theories support this placement. We discuss some short-term challenges post-clerkship Step 1 schools may encounter due to the proposed timing of the change in scoring, which creates three unique scenarios for learners that can introduce inequity in the system and provoke anxiety. We review outcomes of potentially heightened importance when Step 1 is P/F, including lower clinical subject exam scores in some clerkships, lower failure rates on Step 1 and stable Step 2 Clinical Knowledge scores with implications for the residency match. We outline the future potential for performance-based time-variable Step 1 study periods that are facilitated by post-clerkship placement of the exam. Finally, we discuss opportunities to achieve the goal of enhancing student well-being, which was a major rationale for eliminating the three-digit score.

Exploring Students' Perspectives on Well-Being and the Change of United States Medical Licensing Examination Step 1 to Pass/Fail.

PHENOMENON: In February 2020, the Federation of State Medical Boards and National Board of Medical Examiners announced that Step 1 of the United States Medical Licensing Examination would transition from a three-digit numerical score to a pass/fail outcome. While several opinion pieces have been authored on the potential implications of this change, no study has formally assessed the student voice. The purpose of this study is to explore medical students' perspectives of a pass/fail Step 1, with an emphasis on how this transition will impact their well-being. Approach: We conducted virtual focus groups from May 2020-June 2020 with first- and second-year medical students from six institutions (n = 30). We analyzed focus group content following the inductive and iterative constructivist approach to produce a thematic analysis. Findings: Participants included females (50%), males (47%), and one non-binary student. The majority were Caucasian (57%), followed by Asian (27%), African American (10%), and Hispanic or Latino/a (7%). Overall, students were confused regarding the decision to transition Step 1 to a pass/fail outcome. They expressed anxiety over the uncertainty of how a pass/fail Step 1 may impact future residency applications and pressure to re-allocate time and resources to other academic pursuits that would make them competitive. Students explicitly stated skepticism or disbelief that a pass/fail Step 1 would improve their well-being. Insights: While the decision to make Step 1 pass/fail was in part intended to decrease stress associated with performance on a single high-stakes exam designed for licensing purposes, it has led to increased worries for students, and secondary, unanticipated consequences remain to be seen. In this new setting, it will be imperative to provide clarity regarding the metrics used to evaluate students and to incorporate their perspectives in future policy changes.